Different Context, Different Query Segment. Same Key Performance Indicators?

Recently, I’ve received a report at work, where the users who came from organic search were classified as a not-so-well engaged as the ones who landing from other traffic sources like social media, direct access, or referring sites. The conclusions of this report was based in the default web analytics metrics (page views/visit, time on site, bounce rate) where each traffic source was treated as a segment.

Immediately, some questions started filling my mind: Is it right to call a user as less engaged just because Google Analytics’ data said that they have higher bounce rates, lower time on site, and shallow navigation? Is it right to use same parameters as quality indicators to measure different segments? And if this user is someone who has developed the habit of visiting a certain page several times a day to stay up-to-date with the latest news? Isn’t he a “an engaged user”?

In a context of use like that, the visits frequently shows bounces at nearly 100%, a time on site up to one minute, and a shallower dept of navigation. But these kind of users literally are someone who really loves that page. He visits it all day long -sometimes at night- and possibly every time when he goes on the internet or just want to know what is happening in the world. He just want to consume on-demand news headlines. Sometimes, when he really likes of one story, he share it on his preferred social media.

But it doesn’t stops here. That user has another behaviour pattern to fit within that same context of use. When he follows the unfolding of some awesome story, he has the habit of — beyond visiting a favorite hard news page — to hang out on Google and search for some more pieces of that story with the intent of get complementary information about it. As above, when that user does this, he’s under the same context of use of someone who visits a hard news homepage. And at this moment, he could be reading some complementary information for the story on the pages of your website. And if he does it, he will leave closely similar footprints that he left at your favourite hard news website. That is: bounces nearly at 100%, a time on site up to one minute, and a more shallow depth of navigation are all possible.

Is this why I felt so strange about the report I received which classified the users who came from search sites as less engaged than those came from all other traffic sources. In other words, it sounds that, after reading the data file, they’ve concluded something like: “Yes, the search sites are bringing us lots of visits, more than all other traffic sources, but that visitors have a kind of low quality, confirmed by the Google Analytics numbers.”

Comparison between three different traffic sources:

Could the data above mean that the users who came from organic search are less engaged than those landing from other traffic sources?

But I was saved by two great authors from that ocean of doubt. First, I reviewed Avinash Kaushik’s warning all at MozCon 2011 about the importance of segmentation. A minute after that, Petter Morville’s book Search Patterns came to my mind. I remembered reading a page showing that each query has its own context of use.

Since this time, I’ve started to thinking about doing a drill-down in organic traffic sources to improve the details of this segment by creating sub-segments based on context of use. As a subsequent act, I established customized key performance indicators (KPIs) to better fit with each context of use based segments. In practice terms, I`ve joined Kaushik’s and Morville’s thinking in trying to serve a more SEO-friendly report. 😛

Different context, different query, same metrics? Use different key performance indicators.

Not everyone knows that the navigation patterns from users accessing a given website from search can be pretty different, according to their intent and the context of site use. Both intent and context of use are intrinsic in the query – all you need to do is uncover them in the search log files.

There are lots of attributes that can be used to do that differentiation on a search log file like the presence of abstract, concrete, or experiential terms; the number of terms that composes a query; the presence of adjectives or superlatives in that query; the finality of the search – that can be a extrinsic or a intrinsic activity within the Internet; and so on. However, one of more determinant characteristics that has a big weight in behaviour patterns is the search’s context of use. The context of use can determine if a user is going to do a broad, medium, or narrow search. And it will determine too, if he will develop a known-item search or a exploratory search behaviour pattern.

But, what does it mean? It means that in some cases or context of use, engagement and user satisfaction can be a bounce rate or a few time on a certain page. For instance, you have to expect different behaviours between people who have searched for “recipes” and who have searched for “recipes for mushroom” or, yet, “recipes containing black morel mushroom.” (I’ve changed the examples just to show how flexible that approach is.)

Each region of a website structure has a different missions in terms of answering a query

Picture 2: more central regions answer for branded and broad query, more peripheral regions answer for narrow query, and midway regions are responsible for answer for medium queries

As was said above, the behaviour you have expect from someone that comes to your website from a search engine varies according with the intent of each user in a given context of use. When you read each search log, it is quite easy to discover the intent of the search. You can identify very quickly if the user did a broad or narrow query, if he used a brand name, if he has used adjectives or superlatives, if your intent is to do something essentially intrinsic at the Internet or not.

By reading each search logs, as a lot of authors – such as John Battele – has been telling along time, you can read the mind of your user. Ow, sounds really good for who works with user experience. 😛 By that way, grounded by a search log segmentation, you will know in which cases a deep navigation, a bounce rate, or a small time-spent is good or not.

How to do

However, that issue can be a nightmare if your mission involves dealing with large search log files. In a case like that, how can you create segments that can show you clues to present this kind of data from large databases? Is there any pattern that can be recognizable in the ocean of search logs and could tell us something about user intent? The answer is yes, fewer but efficient.

One of the only things we know about the user’s behaviour, after he has performed a web search, is that it will be quite different depending if your query was broad or narrow.

Unless you are a programming ninja with superpower to develop a Named Entities Recognition software, the unique way for a mortal human to identify grammatical issues at search log files is to read one by one and attach the references above using a spreadsheet. And this can be insane in some cases. But using Google Analytics, you can do a lot just segmenting by the length of the queries and on branded searches. It will not be extremely precise, but can fit the role.

So, let’s take a look at what you have to do to increase user engagement and satisfaction if he did a search before reaching your website.

1. isolate the branded searches
2. take the measurements for a 1-2 term query
3. take the measurements for a 3 terms query
4. take the measurements for queries from 4 or more terms per query

For the segments #1 and #2, you have expect more deep navigation. For the segments #3 and #4, you can expect more shallow navigation. So, let’s go to the practice.

You can see it happening right here in the image below. In this example, extracted from a Brazilian website named “Guia do Estudante”, is relatively easy to figure out that the numbers changes dramatically since the number of terms in each entrance query varies.

Comparison between branded, broad, medium, and narrow query from the organic entrance sources.

Picture 3: Who has medium or lower engagement now?

As you read above, people that perform branded, broad, medium, and/or narrow searches tend to develop different navigational patterns. And that is why you have to use different KPIs to take the measurements of user engagement or satisfaction.

It is more evident when you assume that the more terms a query has, the more the user knows about what he is searching for. Where are the users with less engagement now?

Now let us do the comparison between the segmented query data above within organic traffic without any segmentation. It is quite easy to perceive that almost a half of total users (around 500,000 visits) who came from the search sites arrive at this website after having done a narrow query, composed of more than four words. The other 1,200,000 visits performed navigational, broad, or medium queries. Now I will ask you to make your own conclusions about those numbers. Which of these users are more or less engaged?

Let’s redefine the navigation engagement qualitative indicators for websites whose revenue is based on a CPM business model.

I took away three important lessons from this experience:

1. All key performance indicators must be oriented to a context of use.
2. The only metric that can disqualify a user who landed at a website after doing a non-branded narrow query is the time on page.
3. The nature of the product under the metrics have be considered too at the moment of setting up a new KPI.

Does it make sense to you? I hope yes! How many behaviour patterns and segments can you use to indicate the quality of your organic visitors who have performed a search query on a search site before reaching your website? 😛